There is a profound need for innovative technology and operations that can defend networks against the growing complexity of network threats and insider attacks. Networks have become an integral part of a wide range of activities including business processes, government operations, and the national power grid. Within this net-centric environment, threats have evolved to be distributed, decentralized, and adaptable, operating over multiple time periods performing data exfiltration, denial of service, and phishing.
Network Intrusion Detection Systems (IDS) offer the ability to deter threats on networks and have traditionally taken the form of firewalls, antivirus software, spyware detection software, and a signature-based detection systems like Snort®. Network IDS systems can be categorized into misuse detection and anomaly detection systems. Anomaly detection capabilities are driven by the normative specification of user and system behaviors on a network that is considered operationally normal and non-threatening. Based on this normative specification, abnormal behaviors are identified by observing deviations from the established normal behavioral patterns. A prior knowledge of threats and their behaviors does not need to be specified before new types of threats can be detected.
Misuse detection and contrary anomaly detection are driven by the specification of the abnormal structural patterns and/or behaviors and is captured in terms of signatures. These systems apply pattern-matching methodologies on live data and known attack signatures to generate alarms. Generally speaking, these systems are deployed and operated in a single location, or single source, and do not match current complex distributed threat models operating well within the noise of everyday traffic.
Because the fundamental detection models that support defensive strategies have not evolved to match current threats, and are still “point source,” we are unable to deter and/or anticipate attacks, making us vulnerable to attacks. Attacks on networks can lead to losses of capital, time, reputation, and intellectual property. Effective network monitoring that is built on a distributed defense model is needed to mitigate current complex threats.
Commercial research network defense technologies, and research and development programs, today fall into two categories: misuse detection and anomaly detection systems. These alert-centric technologies were created to protect the perimeter, deriving sensor events from a single ingress point of an enterprise network. In order to scale to the needs of botnet detection, correlation capabilities are needed to bring alerts together to define botnet behaviors derived from multiple ingress points and then correlate the alerts to a single botnet threat. BotHunter contains such a correlation engine needed to bring together events. Another system, Worminator, addresses the need for sharing alerts between detection nodes using a collaborative distributed intrusion methodology system. These systems connect communication events associated with distributed threats, like botnets, based on modeling the patterns between hosts within a network. Because of the variability in communication patterns, a system needs to employ parallel detection strategies to detect botnets.
There has been a wide range of anomaly detection behavioral models created in the past as, illustrated in the taxonomy shown in FIG. 2. These behavioral models are broken down into two broad types: a learnt model and a specification model. The learnt model employs unsupervised learning methods to discover anomalies without prior knowledge, while the specification model requires a description of the anomaly to detect known threats.